Response Bias in Computerised Tests

Choosing a response format is a problem when designing computerised tests. Programmers often solve this problem by highlighting a response option and allowing the user to select this option or choose another by using the cursor keys. It is possible that such well-meaning attempts to make computers more userfriendly may heighten response style tendencies. The two experiments to be reported in this study were designed to test the effect of cursor positioning in personality and ability tests. In the first experiment, sixty two Participants were randomly assigned to two groups and asked to complete computerised versions of Forms A and B of the Eysenck Personality Inventory. The position of the highlight was manipulated to produce a two-by-two factorial design with repeated measures on one factor. Results showed that there was evidence of a cursor effect but that it was confined to one of the conditions in the design. Experiment two employed the same sort of manipulation in a computerised version of the ACER Word Knowledge Test with another sample of 36 participants. Here, no evidence was found for any effect of cursor positioning. Overall, it was concluded that the highlighting technique itself is unlikely to have any effect on measures of cognitive ability but that it may have some effect in the more subjective personality testing domain. Response Bias in Computerised Tests 3 Response Bias in Computerised Tests Testing was one of the earliest areas within psychology to benefit from computer applications with standardised, objective-type personality tests being particularly well-suited to automation (Bartram & Bayliss, 1984; Butcher, 1985; Butcher, Keller & Bacon, 1985). Initially, interest focussed on automated scoring but later expanded to include the computerised administration of existing penciland-paper tests (Bartram & Bayliss, 1984; Booth-Kewley, S., Edwards, J.E., & Rosenfeld, P., Butcher, 1985). Currently, almost every facet of personality testing has been computerised, from test design and development, through item generation and analysis, to test interpretation and report generation (Roid, 1986). In a typical computerised test presentation, individual questions or stimuli are presented on a video display unit (VDU) attached to the computer, a set of limited responses is offered, and test-takers record their selected response via a keyboard or some other interface (Butcher, Keller & Bacon, 1985). The advantages of this form of administration over conventional administration are well-documented in several reviews (e.g., Bartram & Bayliss, 1984; Erdman, Klein & Greist, 1985; Butcher, Keller & Bacon, 1985). The move from paper-and-pencil tests to computer-based formats, however, represents a major shift in the way tests are administered and it is important that research is conducted to check the equivalence of the two methods. Work has already started in this area, especially on the equivalence of paper-andpencil versus computerised presentation. Reviews of these studies report conflicting findings, with many uncontrolled variables influencing the outcomes (e.g., Burke & Normand, 1987; Webster & Compeau, 1996). Within the computerised testing domain itself, there is also scope for research. One area which is currently under-researched is the possible effects of computerisation on response sets. Despite claims that there is no bias that is unique to computer administration (Moreland, 1986) and that the effects of response sets may even be minimised or better assessed using computer presentation (Ben-Porath & Butcher, 1986; Davis & Cowles, 1989; Huba, 1986), it may be the case that in our attempts to make computers user friendly, we have introduced techniques which favour response biases not present in paper-and-pencil versions of that same test. Among the studies that have looked at aspects of this topic, there is some evidence indicating the existence of unique computer response sets. It has been noted, for example, that different computer versions of the same test can produce significant differences in scores, apparently related to the stimulus or procedural differences in the two versions (Hedlund, 1988). It has also been found that subtle differences between test versions may produce large differences in scores including, for example, an option which must be actively accepted or rejected on the computer, but which may be scored by the absence of action in conventional administrations (Moreland, 1986). Most (1987) also proposed that subtle differences in presentation, such as flashing a confirmation of the selected response, can have specific, anxiety-arousing effects. Referring to psychological tests, several authors (Duthie, 1984; Johnson and Johnson, 1981) have suggested that the human-computer interface is a likely source of increases in error patterns or response sets. More specifically, Wood (1984 cited in Wilson, Genco & Yager, 1985) has proposed that test scores may be influenced by the format of information on a VDU. Response Bias in Computerised Tests 4 Highlighting is one very popular, apparently inert technique that is often employed on the VDU interface to aid in the selection of response alternatives (Galitz, 1987). For example, with a simple Yes/No response choice, both "Yes" and "No" may be shown in text on the screen. However, it is common for one response to be highlighted initially. For example, "Yes" may be surrounded by a different colour or inverse-video, indicating to the test-taker that the "Yes" response is currently selected, and that some further action is necessary to select "No". It is possible that the initial position of the highlight over one response alternative might influence Participants to respond in that direction. Liefeld (1988) dubbed this the "initial cursor placement effect". A technique introduced to improve the human-computer interface may inadvertently prove to be a source of response bias. The most important ramification of such a response set would be the generation of a test score which was inaccurate and which precipitated unwarranted action on the part of the psychologist. An example of a computerised test that uses such a response format is the Self-Directed Search: Computer Version 3.0 (1989). In the Self-Estimates section of this computerised version of a popular career selection inventory, there are actually 7 possible responses, all of which are listed in column format on the screen. The cursor rests over the number "4", which is the middle of the response range and if the subject is not happy with this reponse, he/she must use the cursor keys to move the highlight to the desired response before pressing the return key. The question is: does the use of the highlight in this fashion predispose the subject to accept the default response, "4"? The present study set out to test the effect of highlighting on the frequency with which a highlighted response was selected in comparison with nonhighlighted response options. Two separate experiments were conducted, each involving a different type of psychological test. In the first experiment, a wellknown personality test was adapted for computer presentation. The second experiment used a test of cognitive ability.